Image harmonization for deep learning model optimization

ABSTRACT

Techniques are described for optimizing deep learning model performance using image harmonization as a pre-processing step. According to an embodiment, a method comprises decomposing, by a system operatively coupled to a processor, an input image into sub-images. The method further comprises harmonizing the sub-images with corresponding reference sub-images of at least one reference image based on two or more different statistical values respectively calculated for the sub-images and the corresponding reference-sub images, resulting in transformation of the sub-images into modified sub-images images. In some implementations, the modified sub-images can be combined into a harmonized image having a more similar appearance to the at least one reference image relative to the input image. In other implementations, harmonized images and/or modified sub-images generated using these techniques can be used as ground-truth training samples for training one or more deep learning model to transform input images with appearance variations into harmonized images.

TECHNICAL FIELD

This application relates to image harmonization techniques for deeplearning model optimization.

BACKGROUND

Advancements in artificial intelligence (AI) and machine learning (ML)technologies, such as deep neural networks (DNN)s, have led to thedevelopment of AI/ML models that have shown impressive performance inmedical image processing and analysis tasks like diagnosis, organsegmentation, anomaly detection, image reconstruction, and so on. Mostoften, these models are trained on images from a specific source domain.When applied to images that vary in appearance from the source domainimages due to various factors (e.g., image capture protocol, dose usage,exposure setting, photon receiving materials, field-of-view (FOV),demography, contrast vs. non-contrast, etc.), model performancedegradation is often observed. It is difficult and costly to adapt thesemodels to accurately perform on images from other domains. Accordingly,efficient and effective techniques for maintaining or improving modelperformance on images with different appearance variations relative tothe original training images are needed.

SUMMARY

The following presents a summary to provide a basic understanding of oneor more embodiments of the invention. This summary is not intended toidentify key or critical elements or delineate any scope of thedifferent embodiments or any scope of the claims. Its sole purpose is topresent concepts in a simplified form as a prelude to the more detaileddescription that is presented later. In one or more embodimentsdescribed herein, systems, computer-implemented methods, apparatusand/or computer program products are described that provide imageharmonization techniques for deep learning model optimization.

According to an embodiment, a system is provided that comprises a memorythat stores computer executable components, and a processor thatexecutes the computer executable components stored in the memory. Thecomputer executable components can comprise an image decompositioncomponent that decomposes an input image into sub-images, and aharmonization component that harmonizes the sub-images withcorresponding reference sub-images of at least one reference image basedon two or more different characteristics respectively calculated for thesub-images and the corresponding reference-sub images, resulting intransformation of the sub-images into modified sub-images images. Insome implementations, the computer executable components can furthercomprise a reconstruction component that combines the modifiedsub-images into a harmonized image, the harmonized image having a moresimilar appearance to the at least one reference image relative to theinput image.

In various implementations, the sub-images comprise energy band imagesand the reference sub-images comprise corresponding reference energyband images. With these implementations, wherein the two or moredifferent characteristics can comprise statistical values selected froma group consisting of: means of the energy band images and thecorresponding reference energy band images, standard deviations of theenergy band images, percentiles of the energy band images, andhistograms of the energy band images.

The computer executable computer executable components further comprisea reference image selection component that selects the at least onereference image from candidate reference images based on a degree ofsimilarity between a first feature vector for the input image and asecond feature vector of each reference image. In some implementations,the first feature vector and the second feature vector are respectivelybased on at least one statistical value of the two or more differentstatistical values. For example, in implementations in which thesub-images comprise energy band images and the reference sub-imagescomprise corresponding reference energy band images, the feature vectorscan be based one or more statistical values such as but not limited to:means of the energy band images, standard deviations of the energy bandimages, percentiles of the energy band images, and histograms of theenergy band images. Additionally, or alternatively, the first featurevector and the second feature vector can be respectively based on outputfeatures generated based on application of a deep learning neuralnetwork model to the input image and the at least one reference image,wherein the deep learning neural network model was trained on a corpusof images related to the input image and the at least one referenceimage.

In some implementations, the computer executable components can furthercomprise a vectorization component that generates feature vectors forcandidate reference images, and reference image set generation componentthat selects a subset of the candidate reference images based ondifferences between the feature vectors. With these implementations, thereference image selection component can select the at least onereference image from the subset. For example, in one implementation, thecomputer executable components further can comprise a clusteringcomponent that clusters the candidate reference images into differentgroups based on the differences between the feature vectors, and thereference image set generation component can select one candidatereference image from each of the different groups for inclusion in thesubset.

In one or more additional implementations, the computer executablecomponents can further comprise a training component that uses theharmonized images as ground-truth training samples to facilitatetraining an image harmonization model to transform the input image intoa harmonized image that has a more similar appearance to the at leastone reference image relative to the new input image. The imageharmonization model can comprise one or more machine learning models,such as deep learning neural network models and the like. For example,the image harmonization model can comprise a plurality of sub-imageharmonization models that respectively transform new sub-imagesdecomposed from the new input image into new modified sub-images. Withthese implementations, the reconstruction component can combine the newsub-images to generate the harmonized image.

In another embodiment, a system is provided that comprises a memory thatstores computer executable components, and a processor that executes thecomputer executable components stored in the memory. The computerexecutable components can comprise decomposition component thatdecomposes training images into sub-images, and a harmonizationcomponent that harmonizes the sub-images with corresponding referencesub-images of reference images, resulting in transformation of thesub-images into modified sub-images. The training images can compriseimages with different appearance variations representative of imagesfrom different domains. The computer executable components furthercomprise a reconstruction component that combines groups of the modifiedsub-images associated with respective training images of the trainingimages to generate ground-truth harmonized images for the respectivetraining images. The computer executable components further comprise atraining component that trains a full-image harmonization model (i.e.,an image harmonization model) to transform the training images into theground-truth harmonized images. The computer executable components canfurther comprise a model application component that applies the(trained) full-image harmonization model to a new input image totransform the new input image into a new harmonized image that has amore similar appearance to the reference images relative to the newinput image.

Additionally, or alternatively, another system is provided thatcomprises a memory that stores computer executable components, and aprocessor that executes the computer executable components stored in thememory. The computer executable components can comprise decompositioncomponent that decomposes training images into sub-images, and aharmonization component that harmonizes the sub-images withcorresponding reference sub-images of reference images, resulting intransformation of the sub-images into modified sub-images. In this case,the harmonized images can be images transformed using energy-bandharmonization or images from the same imaging setting. The computerexecutable components further comprise a training component that trainssub-image harmonization models to transform the sub-images into themodified sub-images. The training images can comprise images withdifferent appearance variations representative of images from differentdomains. The computer executable components can further comprise a modelapplication component that applies the sub-image harmonization models totransform new sub-images decomposed from an input image into newmodified sub-images, and a reconstruction component that combines thenew modified sub-images to generate a harmonized image for the inputimage, the harmonized image having a more similar appearance to thereference images relative to the input image.

In some embodiments, elements described in the disclosed systems can beembodied in different forms such as a computer-implemented method, acomputer program product, or another form.

DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a block diagram of an example, non-limiting systemthat facilitates image harmonization for deep learning modeloptimization in accordance with one or more embodiments of the disclosedsubject matter.

FIG. 2 illustrates decomposition of an image into different energy bandimages in accordance with one or more embodiments of the disclosedsubject matter.

FIG. 3 presents a flow diagram of an example image harmonization processin accordance with one or more embodiments of the disclosed subjectmatter.

FIG. 4 presents chest X-ray images before and after image harmonizationin accordance with one or more embodiments of the disclosed subjectmatter.

FIG. 5 presents magnetic resonance (MR) scan images before and afterimage harmonization in accordance with one or more embodiments of thedisclosed subject matter.

FIG. 6 presents X-ray images of different anatomical parts before andafter image harmonization in accordance with one or more embodiments ofthe disclosed subject matter.

FIG. 7 illustrates an example, high-level flow diagram of acomputer-implemented process for image harmonization in accordance withone or more embodiments of the disclosed subject matter.

FIG. 8 illustrates a block diagram of another example, non-limitingsystem that facilitates image harmonization for deep learning modeloptimization in accordance with one or more embodiments of the disclosedsubject matter.

FIG. 9 illustrates an example, high-level flow diagram of anothercomputer-implemented process for image harmonization in accordance withone or more embodiments of the disclosed subject matter.

FIG. 10 illustrates an example, high-level flow diagram of anothercomputer-implemented process for image harmonization in accordance withone or more embodiments of the disclosed subject matter.

FIG. 11 illustrates a block diagram of another example, non-limitingsystem that facilitates image harmonization for deep learning modeloptimization in accordance with one or more embodiments of the disclosedsubject matter.

FIG. 12 illustrates training and usage of a full-image harmonizationmodel in accordance with one or more embodiments of the disclosedsubject matter.

FIG. 13A illustrates training of sub-image harmonization models forimage harmonization in accordance with one or more embodiments of thedisclosed subject matter.

FIG. 13B illustrates application of sub-image harmonization models forimage harmonization in accordance with one or more embodiments of thedisclosed subject matter.

FIG. 14 illustrates an example, high-level flow diagram of acomputer-implemented process for image harmonization using a full-imageharmonization model in accordance with one or more embodiments of thedisclosed subject matter.

FIG. 15 illustrates an example, high-level flow diagram of acomputer-implemented process for developing a full-image harmonizationmodel in accordance with one or more embodiments of the disclosedsubject matter.

FIG. 16 illustrates an example, high-level flow diagram of acomputer-implemented process for image harmonization using sub-imageharmonization models in accordance with one or more embodiments of thedisclosed subject matter.

FIG. 17 illustrates a block diagram of an example, non-limitingoperating environment in which one or more embodiments described hereincan be facilitated.

DETAILED DESCRIPTION

The following detailed description is merely illustrative and is notintended to limit embodiments and/or application or uses of embodiments.Furthermore, there is no intention to be bound by any expressed orimplied information presented in the preceding Summary section or in theDetailed Description section.

The disclosed subject matter is directed to systems,computer-implemented methods, apparatus and/or computer program productsthat provide various image harmonization techniques to facilitateoptimizing the performance of AI/ML-based image processing models onimages that vary in appearance relative to the training images. In thecontext of the disclosed subject matter, image harmonization refers tothe adaptation of a given image to appear more similar to one or morereference images that are representative of the training images used totrain a particular image processing model. For example, the one or morereference images can include one or more of the training images and/orone or more images that correspond to the training images The disclosedimage harmonization techniques are independent of the trained model andcorrect the model prediction for an image from a different domainrelative to the training images as a pre-processing step. In thisregard, the disclosed image harmonization techniques can be applied tofacilitate optimizing the performance of various types of imageprocessing models configured to perform a variety of inferencing tasks.

In one or more non-limiting embodiments, the image processing models caninclude medical image processing models based on deep learning that areconfigured to perform various medical related inferencing tasks onmedical images, such as image-based diagnosis, organ segmentation,anomaly detection, image reconstruction, and the like. For example,various deep-learning-based AI models have been proposed to interpretand prioritize chest X-ray/radiography (CXR) images, which are the mostcommonly used modalitiy for screening and diagnosing various lungdiseases such as pneumonia, lung cancer, tuberculosis and lung tissuescarring. However, quite often these regulated products are onlyauthorized for usage with images captured using a specific chest X-raysystem, as the appearance of C×R images can vary between different X-raysystems due to dose usage, exposure setting, photon receiving materials,and various other factors. These regulated AI models can be verysensitive to the change in images caused by these imaging factors whichis also the case in general for deep-learning networks. Often a decreasein performance can be observed on image datasets which are differentthan the training dataset due to even minor appearance variations.

Thus, in the context of medical image processing model optimization, thedisclosed image harmonization techniques can be used to adapt medicalimages that vary in appearance relative to the model training images tobe more similar in appearance to the training images prior to input intothe model. For example, the variational medical images can comprisemedical images of a same type as the training images (e.g., C×R images)that vary in appearance to the training images owing to differences inthe capture protocol/system used (e.g., which can vary between medicalcenters) or other imaging factors. The disclosed image harmonizationtechniques can also be applied to harmonize image datasets for AI/MLimage analysis/processing models configured to perform variousinferencing tasks on non-medical images.

In accordance with one or more embodiments, the disclosed imageharmonization techniques involve decomposing an original image to beharmonized with a particular model's training dataset into sub-imagesand harmonizing the sub-images with corresponding reference sub-imagesdecomposed from one or more reference images. In particular, thesub-images can be harmonized with the corresponding reference sub-imagesby adapting or modifying the sub-images to appear more similar to thecorresponding reference sub-images, resulting in modified sub-images forthe original image. In various embodiment the harmonization process caninvolve changing one or more features of each sub-image to make themmore similar to the corresponding features of a corresponding referencesub-image. The modified sub-images can then be re-combined to generate areconstructed, harmonized image that is a modified version of theoriginal image having a more similar visual appearance to the one ormore reference images relative to the original image.

In various embodiments, the decomposition involves decomposing theoriginal image and the one or more reference images into differentenergy band images with different frequency ranges using a noveldecomposition process that generates the sub-band images from a low passsignal. After the original image and the one or more reference imageshave been respectively decomposed into energy band images, each energyband of the original image can be harmonized with one or morecorresponding reference energy band images by making one or morestatistics of each energy band image similar to those of the one or morecorresponding reference energy band images. For example, variousstatistical measures can be calculated or generated for each energy bandimage based on its energy band image intensity values, including but notlimited to: the mean of its energy band image intensity values, thestandard deviation of its energy band image intensity values, thepercentiles of its energy band image intensity values, and a histogramof its energy band image intensity values. In this regard, harmonizationof an energy band image for the original image with a correspondingreference energy band image can comprise modifying (e.g., with respectto a defined degree of change) intensity values of the energy band imagesuch that the statistical measures of the energy band image intensityvalues are more similar to those of the corresponding reference energyband image.

In another embodiment, the decomposition of the original input image andthe one or more reference images into sub-images can comprise segmentingthe image into two or more different regions. For example, as applied tomedical images, a medical image can be segmented into different regions(e.g. organs) of interest using a pre-trained segmentation model. Inthis regard, each segmented region can be considered a sub-image andwill have different features (e.g., with respect to appearance featuresand/or spatial features). According to this embodiment, respectivesegmented regions for the original image can be harmonized withcorresponding segmented regions of one or more reference images bymaking one or more features of the original image segments more similarto the corresponding features of the corresponding reference imagesegments. The modified segments of the original image can then berecombined to generate a harmonized image.

In various additional embodiments, the harmonized images and/or modifiedsub-images generated using the techniques described above can also beused as ground-truth training samples for training one or more ML/AImodels to transform input images with appearance variations intoharmonized images (e.g., harmonized with an image processing model'straining dataset) prior to input into the image processing model. SuchML/AI models are referred to herein as image harmonization models. Forexample, the one or more image harmonization models can comprise one ormore deep learning models, such an autoencoder, a generative adversarialautoencoder, a generative adversarial network (GAN), or the like. Withthese embodiments, once trained, the one or more image harmonizationmodels can be used to transform new input images with different featurevariations into harmonized images that have a more similar appearance tothe original source training images.

For example, in some embodiments, the above described techniques forgenerating harmonized images can be applied to a set of imageharmonization model training images having different appearancevariations to generate harmonized images for each of the trainingimages. An image harmonization model can then be trained to transformthe training images into the harmonized images. Additional trainingimages without ground-truth harmonized versions can also be added to thetraining dataset. Once the image harmonization model has been trained,the image harmonization model can be applied to transform new inputimages with feature variations relative to the original model trainingimages into harmonized images.

Additionally, or alternatively, the groups of sub-image/modifiedsub-image pairs generated for each of the training images can be used totrain separate sub-image harmonization models to map the sub-images tothe corresponding modified sub-images. For example, assume thedecomposition process decomposes each input image into a first passenergy band image, a second pass energy band image, a third pass energyband image, and so on up to a defined number (i) of passes. According tothis example, a first sub-image harmonization model can be trained totransform the first pass energy band images into first modifiedsub-images, a second sub-image harmonization model can be trained totransform the second pass energy band images into second modifiedsub-images, and so on. Once trained, the sub-image harmonization modelscan be respectively applied to new first pass, second pass, third pass,etc. sub-images decomposed from a new input image to generate modifiedsub-images for the new input image. The modified sub-images can then berecombined to generate a harmonized image.

The disclosed techniques for developing and applying one or more imageharmonization models provide a highly efficient end-to-end process forimage harmonization as the mechanism for generating the ground-truthtraining data is automated. In this regard, one major limitation inML/AI model development is the availability of ground-truth trainingdata, which most often requires manual annotation, a tedious and costlyendeavor. With the disclosed techniques, the ground-truth training datacan be generated without any manual intervention, thus providing forlimitless, low-cost, and efficient ground-truth training datageneration. In addition, the usage of a trained image harmonizationmodel to transform input images into harmonized images reduces theoverall processing time for image harmonization relative to abovedescribed decomposition, sub-image harmonization and reconstructionprocess described above.

The disclosed subject matter further provides tools for enhancing theaccuracy and effectiveness of the disclosed image harmonizationtechniques by optimizing reference image selection. In this regard, insome embodiments, a same, single, pre-selected reference image can beused to transform all variational images into harmonized images forinput into a particular image processing model. In other embodimentsmultiple (e.g., two or more) reference images can be used. According tothese embodiments, the sub-images decomposed from an input image can beharmonized with corresponding reference sub-images of the multiplereference images using a weighting scheme for the reference sub-imagesdetermined based on degrees of similarity between the input image andthe respective reference images.

In various implementations, the degrees of similarly can be determinedbased on comparison of feature vectors respectively generated for theinput image and the reference images. In some implementations, thefeature vectors can be generated based on one or more of the statisticalmeasures that will be used to harmonize the sub-images with thecorresponding reference sub-images. For example, in implementations inwhich the images are decomposed into energy band images, the featurevectors can be generated based on one or more statistics of theirrespective energy band images (e.g., the means of the energy band imageintensity values for each energy-band image, the standard deviations ofthe energy band image intensity values for each energy-band image, thepercentiles of the energy band image intensity values for eachenergy-band image, etc.). Additionally, or alternatively, the featurevectors can be computed from the output features of one or more layers(e.g., the fully connected layer) of a pretrained network trained on acorpus of similar images (e.g., using an ImageNet dataset or the like).The feature vectors can also represent other image-based features (e.g.,dose usage, exposure setting, photon receiving materials, FOV,demography, contrast vs. non-contrast, etc.) and/or non-image basedfeatures (e.g., patient specific features such as patient demographics,patient medical history, etc.) included in metadata associated with therespective images.

In some embodiments, the multiple reference images can include a selectsubset of the original model training images that provide arepresentative subset of the different types of variational imagesincluded in the training dataset. For example, in one or moreimplementations, feature vectors can be generated for all (or a smallersubset) of the training images. The training images can then beclustered into different groups of related images based on similaritiesand differences between their feature vectors (e.g., using a suitableclustering algorithm such as k-means clustering or the like). One ormore representative reference images can then be selected from each (orin some implementations one or more) group to generate a set ofreference images that are used for harmonization using a weightingscheme based on the degrees of similarity between the input image andthe respective references images included in the set.

Additionally, or alternatively, a single reference image can be selectedfrom amongst the set of reference images or the entire set of trainingimages for each input image to be harmonized based on feature vectorsimilarity. In this regard, the feature vector for the input image canbe compared with the feature vectors of all of the candidate referenceimages to select a single candidate reference image that is the bestmatch based on its feature vector having the greatest degree ofsimilarity to the feature vector for the input image. This process canbe performed for each input image such that the reference image used toharmonize each input image can be tailored to the input image.

The term “image processing model” is used herein to refer to an AI/MLmodel configured to perform an image processing or analysis task onimages. The image processing or analysis task can vary. In variousembodiments, the image processing or analysis task can include, (but isnot limited to): a segmentation task, an image reconstruction task, anobject recognition task, a motion detection task, a video tracking task,an optical flow task, and the like. The image processing modelsdescribed herein can include two-dimensional image processing models(2D) as well as three-dimensional (3D) image processing models. Theimage processing model can employ various types of AI/ML algorithms,including (but not limited to): deep learning models, neural networkmodels, deep neural network models (DNNs), convolutional neural networkmodels (CNNs), and the like.

As used herein, a “medical imaging processing model” refers to an imageprocessing model that is tailored to perform an imageprocessing/analysis task on one or more medical images. For example, themedical imaging processing/analysis task can include (but is not limitedto): organ segmentation, anomaly detection, anatomical featurecharacterization, medical image reconstruction, diagnosis, and the like.The types of medical images processed/analyzed by the medical imageprocessing model can include images captured using various types ofimaging modalities. For example, the medical images can include (but arenot limited to): radiation therapy (RT) images, X-ray images, C×Rimages, digital radiography (DX) X-ray images, X-ray angiography (XA)images, panoramic X-ray (PX) images, computerized tomography (CT)images, mammography (MG) images (including a tomosynthesis device), amagnetic resonance imaging (MRI) images, ultrasound (US) images, colorflow doppler (CD) images, position emission tomography (PET) images,single-photon emissions computed tomography (SPECT) images, nuclearmedicine (NM) images, and the like. The medical images can includetwo-dimensional (2D) images as well as three-dimensional images (3D).

The terms “source domain model”, “source model” “source image processingmodel”, “source domain image processing model” and the like are usedherein interchangeably to refer to an imaging processing model trainedon images from specific domain, referred to herein as the source domain.Images included in the source domain are referred to herein as “sourcedomain images” or “source images.” As applied to image harmonization,the reference image or images are or correspond to source domain images.In this regard, images that vary in appearance from the source domainimages are considered herein as images from a different but similardomain relative to the source domain. These images can comprise imagesof the same “type” as the source domain images, yet that have someappearance variations relative to the source domain images. For example,the appearance variations can be attributed to one different imagingfactors, including but not limited to: image capture protocol, imagecapture modality, capture voltage, sequence intensities, dose usage,exposure setting, photon receiving materials, FOV, demography, contrastvs. non-contrast, and other factors affecting image quality (IQ) orappearance. For example, the source domain images and the variationalimages can comprise a same type of medical image yet captured fromdifferent acquisition sources used at different medical centers. Inanother example, the source domain images and the variational images cancomprise medical images of a same anatomical body part (e.g., a sameorgan), yet differ with respect to capture modality (CT images with andwithout contrast). In another example, the source domain images and thevariational images can vary with respect to different capture voltages.In another example, the source domain images and the variational imagescan include MR images that vary with respect to sequence intensities.

One or more embodiments are now described with reference to thedrawings, wherein like referenced numerals are used to refer to likeelements throughout. In the following description, for purposes ofexplanation, numerous specific details are set forth in order to providea more thorough understanding of the one or more embodiments. It isevident, however, in various cases, that the one or more embodiments canbe practiced without these specific details.

Turning now to the drawings, FIG. 1 illustrates a block diagram of anexample, non-limiting system 100 that facilitates image harmonizationfor deep learning model optimization in accordance with one or moreembodiments of the disclosed subject matter. Embodiments of systemsdescribed herein can include one or more machine-executable componentsembodied within one or more machines (e.g., embodied in one or morecomputer-readable storage media associated with one or more machines).Such components, when executed by the one or more machines (e.g.,processors, computers, computing devices, virtual machines, etc.) cancause the one or more machines to perform the operations described.

For example, system 100 comprises a computing device 104 that caninclude various computer/machine executable components, includingdecomposition component 106, image harmonization component 108 andreconstruction component 110. The computing device 104 can furtherinclude or be operatively coupled to at least one memory 118 and atleast one processor 116. In various embodiments, the at least one memory118 can store executable instructions (e.g., the decomposition component106, the image harmonization component 108, the reconstruction component110, and additional components described herein) that when executed bythe at least one processor 116, facilitate performance of operationsdefined by the executable instructions. The computing device 104 furtherincludes a reference image data source 114 that can store one or morereference images. In other implementations, the one or more referenceimages can be stored in memory 118 or another suitable data structureaccessible to the computing device 104. The computing device 104 canfurther include a device bus 112 that communicatively couples thevarious components of the computing device 104. Examples of saidprocessor 116 and memory 118, as well as other suitable computer orcomputing-based elements, can be found with reference to FIG. 17 withrespect to processing unit 1716 and system memory 1714, and can be usedin connection with implementing one or more of the systems or componentsshown and described in connection with FIG. 1 or other figures disclosedherein.

In some embodiments, system 100 can be deployed using any type ofcomponent, machine, device, facility, apparatus, and/or instrument thatcomprises a processor and/or can be capable of effective and/oroperative communication with a wired and/or wireless network. All suchembodiments are envisioned. For example, the computing device 104 can beor correspond to a server device, a general-purpose computer, aspecial-purpose computer, a tablet computing device, a handheld device,a server class computing machine and/or database, a laptop computer, anotebook computer, a desktop computer, a cellular phone, a smart phone,a consumer appliance and/or instrumentation, an industrial and/orcommercial device, a digital assistant, a multimedia Internet enabledphone, a multimedia player, and/or another type of device.

It should be appreciated that the embodiments of the subject disclosuredepicted in various figures disclosed herein are for illustration only,and as such, the architecture of such embodiments are not limited to thesystems, devices, and/or components depicted therein. In someembodiments, one or more of the components of system 100 and othersystems described herein can be executed by different computing devices(e.g., including virtual machines) separately or in parallel inaccordance with a distributed computing system architecture. System 100can also comprise various additional computer and/or computing-basedelements described herein with reference to operating environment 1700and FIG. 17 . In several embodiments, such computer and/orcomputing-based elements can be used in connection with implementing oneor more of the systems, devices, components, and/or computer-implementedoperations shown and described in connection with FIG. 1 or otherfigures disclosed herein.

The decomposition component 106, image harmonization component 108 andreconstruction component 110 can collectively perform an imageharmonization process that can be used to transform an input image 102into a harmonized image 118. The harmonized image 118 generated inaccordance with this image harmonization process is a modified versionof the input image 102 that has a more similar appearance to one or morereference images (e.g., included in the reference image data source 114)relative to the input image 102. In various embodiments, the one or morereference images can be or correspond to training images from aparticular source domain that were used to train a particular AI/MLimage processing model. Thus, the image harmonization process providedby the decomposition component 106, the image harmonization component108, and the reconstruction component 110 can be used to transform aninput image 102 that has or may have some appearance variations relativeto the training images into a harmonized image 118 that corrects ornormalizes those appearance variations such that the harmonized image118 appears more similar to the training images. Accordingly, the imageharmonization process provided by the decomposition component 106, theimage harmonization component 108, and the reconstruction component 110can be used as a pre-processing step to transform images with appearancevariations relative to an image processing model's training images intoharmonized images prior to input into the image processing model. As aresult, the image processing model can be applied to images from variousdomains other than the source domain with no or minimal performancedegradation.

In accordance with one or more embodiments, the image harmonizationprocess involves decomposing the input image 102 into sub-images by thedecomposition component 106, harmonizing the sub-images withcorresponding reference sub-images decomposed from the one or morereference images to generate modified sub-images by the harmonizationcomponent 108, and recombining the modified sub-images by thereconstruction component 110 to generate the harmonized image 118. Insome embodiments a single reference image can be used for harmonization.For example, the single reference image can serve as a singlerepresentative image that represents the training image dataset. In someimplementations, the single reference image can be pre-selected. Inother implementations, the single reference image can be selected fromamongst candidate reference images based on its degree of similarity tothe input image. In other embodiments, multiple reference images (e.g.,two or more) can be used for harmonization. Additional features andfunctionalities regarding reference image selection and usage ofmultiple reference images are described infra with reference to FIG. 8 .For ease of explanation, the features and functionalities of the system100 are initially described with reference to FIGS. 1-7 in accordancewith embodiments that use a single reference image.

In this regard, the decomposition component 106 can decompose the inputimage 102 into two or more sub-images using one or more decompositionprocesses. In some embodiments, the reference image as provided in thereference image data source 114 can be pre-decomposed into thecorresponding reference sub-images generated using the samedecomposition processed used to decompose the input image 102. In otherimplementations, the decomposition component 106 can decompose both theinput image 102 the reference image (or images) at runtime.

In various embodiments, the decomposition component 106 can decomposethe input image 102 using a novel energy band decomposition process thatcomprises decomposing the input image 102 into different energy bandimages (also referred to as sub-band images). With these embodiments,the same energy band decomposition process is applied to the referenceimage to decompose the reference image into corresponding referenceenergy band images. As used herein, the term “energy band image” (orsub-band image) is used to refer to a frequency domain filtered versionof an image that contains a subset of the spectral components of theoriginal image. In this regard, each energy band image generated fromthe original image has a different frequency band or frequency range anddifferent energy frequency values or intensity values within itsfrequency band/range.

For example, FIG. 2 illustrates decomposition of an input image 202 intodifferent energy band images in accordance with one or more embodimentsof the disclosed subject matter. In various embodiments, the input image202 can be or correspond to input image 102. In this example, the inputimage 202 comprises a C×R image, however, it should be appreciated thatthe input image can comprise other types of medical and non-medicalimages. In the embodiment shown, the input image 202 has been decomposedinto four different energy band images, respectively identified asenergy band image 204, energy band image 206, energy band image 208, andenergy band image 210. Each of the different energy band images arefiltered version of the input image 202 with different frequencyintensities removed.

With reference to FIG. 1 in view of FIG. 2 , in various embodiments, thedecomposition component 106 can decompose an input image (e.g., inputimage 102, input image 202, and the like), into different energy bandimages (e.g., energy band image 204, energy band image 206, energy bandimage 208, and energy band image 210) using a decomposition process thatinvolves generating new energy band images that are high pass bands)bands from a low pass signal band. In this regard, the decompositioncomponent 106 can decompose the original image into different energyband images with different frequency bands, wherein each energy bandimage is filtered from a previously generated low pass signal (e.g.,from a low pass band to a higher pass band). The last low pass signalband can further be included as the last band to cover all frequencybands. In this regard, as expressed mathematically, the decompositioncomponent 106 can decompose the original input image I(x) into differentenergy band images I_(i) with different energy band frequencies inaccordance with Equations 1-3 below. The number (i) of different energyband images can vary, wherein i=1, 2 . . . , B such that the last energyband image is denoted as B.L _(i)(x)=L _(i-1)(x)*G(x;σ _(i)),  Equation 1.

-   -   wherein G is a Gaussian kernel and    -   σ_(i) can be a randomly selected increasing number        I _(i)(x)=L _(i-1)(x)−L _(i)(x) for i=1, . . . ,B−1,  Equation        2.    -   wherein L₀(x)=I(x)        I _(B)(x)=L _(B-1)(x) for i=B   Equation 3.

In accordance with Equations 1-3, the decomposition component 106 canfirst perform a Gaussian calculation (convolution) on the original inputimage I(x) to get image L_(i)(x) which is a smoother version of theoriginal input image in accordance with Equation 1. In this regard,L_(i)(x) is the smoothed image based on the previous image and a definedGaussian kernel. For the first time computation (i.e., the first pass),the previous smoothed image is just the input image. Next, in accordancewith Equation 2, the decomposition component 106 can perform asubtraction between the L_(i-1)(x) and the smooth image from theprevious step L_(i)(x) to get an energy band image I_(i). The resultingenergy band image belongs to a high frequency. As noted, whereinL₀(x)=I(x), from this first past wherein i=1, this first subtractionwill be a subtraction of the first smooth image L_(i)(x) from theoriginal input I(x), to get the first energy band image I₁. As reflectedby Equation 3, the decomposition component 106 can perform theoperations of Equations 1 and 2 multiple times, wherein each time, alower frequency smooth version of the input image generated usingEquation 1 is used for the subtraction to generate the next energyband-image. In this regard, each new energy band image is generated froma low pass energy band image, wherein the new energy band imagegenerated has a higher frequency band relative to the low pass energyband from which it was generated (i.e., from which it was subtracted).

For example, with reference to FIG. 2 , the four different energy bandimages correspond to the resulting images generated through each pass ofEquations 1-3, wherein four passes were performed. In this regard,energy band image 204 can be or correspond to a first pass sub-bandimage (I₁) that comprises first sub-band energy frequencies of the inputimage 202, energy band image 206 can be or correspond to a second passsub-band image (I₂) that comprises second sub-band energy frequencies ofthe input image 202, energy band image 208 can be or correspond to athird pass sub-band image (I₃) that comprises third sub-band energyfrequencies of the input image 202, and energy band image 210 can be orcorrespond to a fourth pass sub-band image (I₄) that comprises fourthsub-band energy frequencies of the input image. Each subsequent energyband image generated through each pass has a higher frequency band rangerelative to the previous image, except for the last one (i.e., energyband image 210), which belongs to the lowest frequency band from whichthe previous images were generated.

The reference image can also be decomposed into different energy bandimages using the same decomposition process described above. In thisregard, the reference image can be decomposed into energy band imageswith the same frequency bands/ranges as the input image. As a result,each of the different reference energy band images can be paired with acorresponding energy band image of the input image. As noted above, insome implementations, the decomposition component 106 can decompose thereference image to generate the reference energy band images in the samemanner as the described for the input image (e.g., using Equations 1-3).

In various embodiments, after the input image (e.g., input image 102,input image 202 and the like) and the reference image have beenrespectively decomposed into energy band images, the harmonizationcomponent 108 can harmonize each energy band image for the input image102 with the corresponding reference energy band image by making one ormore statistics of the energy band image the same or similar those ofthe corresponding reference energy band image. For example, variousstatistical measures can be calculated or generated for each energy bandimage based on its energy band frequencies/intensities, including butnot limited to: the mean of its energy band image intensity values, thestandard deviation of its energy band image intensity values, thepercentiles of its energy band image intensity values, and a histogramof its energy band image intensity values. In this regard, theharmonization component 108 can harmonize each energy band image of theinput image 102 with a corresponding reference energy band image bymodifying (e.g., with respect to a defined degree of change) the energyband frequencies/intensities of the energy band image such that one ormore of the statistical measures of the energy bandfrequencies/intensities are the same or more similar to those of thecorresponding reference energy band image. As a result, the imageharmonization component 108 can transform each energy band image of theinput image 102 into a modified energy band image that is more similarto a corresponding reference energy band image.

For example, FIG. 3 presents a flow diagram of an example imageharmonization process 300 in accordance with one or more embodiments ofthe disclosed subject matter. The image harmonization process 300illustrated in FIG. 3 provides an example embodiment of the disclosedimage harmonization techniques as applied to C×R images andharmonization with energy bands. However, the image harmonizationprocess 300 can be applied to various other types of images using otherdecomposition techniques to generate the sub-images.

With reference to FIG. 3 in view of FIG. 1 , in accordance with imageharmonization process 300, at 301 the decomposition component 106 candecompose an input image 302 into sub-images 304. In one or more exampleimplementations, the input image 302 can be or correspond to a CXR imageto be harmonized with the training image dataset used to train aparticular image processing model. In this regard, the input image 302can comprise a CXR image from a different domain relative to thetraining image dataset. For example, the input image 302 may have beencaptured using a different X-ray system/protocol relative to thetraining images. Regardless of the reason, the input image 302 can haveat least some appearance variations relative to the training images.

In the embodiment shown, the different sub-images are respectivelyidentified as sub-image 304 ₁, sub-image 304 ₂, sub-image 304 ₃ andsub-image 304 ₄. For example, as applied to energy band imagedecomposition, each sub-image can be a different energy band imagegenerated from each pass through Equations 1-3. In this regard,sub-image 304 ₁ can correspond to a first energy band image with a firstfrequency range, sub-image 304 ₂ can correspond to a second energy bandimage with a second frequency range, sub-image 304 ₃ can correspond to athird energy band image with a third frequency range, and sub-image 304₄ can correspond to a fourth energy band image with a fourth energyfrequency range.

Similarly, at 303, the decomposition component 106 can decompose areference image 306 into reference sub-images 308 using the samedecomposition process used for the input image 302. In the embodimentshown, the different reference sub-images are respectively identified asreference sub-image 308 ₁, reference sub-image 308 ₂, referencesub-image 308 ₃ and reference sub-image 308 ₄. For example, as appliedto energy band image decomposition, each of the reference sub-images 308can also be a different energy band image generated from each passthrough Equations 1-3. In this regard, reference sub-image 308 ₁ cancorrespond to a first reference energy band image with the firstfrequency range, reference sub-image 308 ₂ can correspond to a secondreference energy band image with the second frequency range, referencesub-image 308 ₃ can correspond to a third reference energy band imagewith the third frequency range, and reference sub-image 308 ₄ cancorrespond to a fourth reference energy band image with the fourthfrequency range.

At 105, the image harmonization component 108 can harmonize thesub-images 304 with the corresponding reference sub-images 308 togenerate modified sub-images 310. In this regard, the imageharmonization component 108 can harmonize sub-image 304 ₁ with referencesub-image 308 ₁ to generate modified sub-image 310 ₁, harmonizesub-image 304 ₂ with reference sub-image 308 ₂ to generate modifiedsub-image 310 ₂, harmonize sub-image 304 ₃ with reference sub-image 308₃ to generate modified sub-image 310 ₃, and harmonize sub-image 304 ₂with reference sub-image 308 ₂ to generate modified sub-image 310 ₂.

In various embodiments in which the sub-images are energy band images,the image harmonization component 108 can calculate, generate orotherwise determine statistical measurements for each of the sub-imagesand each of the reference sub-images based on their respective energyfrequencies/intensities included within their frequency range/band). Forexample, statistical measurements for each of the sub-images 304 and thereference sub-images 308 can include but are not limited to: the mean ofits energy band image intensity values, the standard deviation of itsenergy band image intensity values, the percentiles of its energy bandimage intensity values, and a histogram of its energy band imageintensity values. The image harmonization component 108 can furtherharmonize each sub-image with its corresponding reference sub-image(e.g., with the same frequency range/band) by modifying theintensities/frequencies of the sub-image to such that its statisticalmeasures are the same or similar to that of the corresponding referencesub-image. For example, the harmonization component 108 can adjust theintensities/frequencies of sub-image 304 ₁ such that one or more of: 1.)the mean of its intensities/frequencies is the same or closer (withrespect to a defined degree of change) to mean of theintensities/frequencies of reference sub-image 308 ₁; 2.) the standarddeviation of its intensities/frequencies is the same or closer (withrespect to a defined degree of change) to the standard deviation of theintensities/frequencies of reference sub-image 308 ₁, 3.) thepercentiles of its intensities/frequencies are the same or closer (withrespect to a defined degree of change) to the percentiles of theintensities/frequencies of reference sub-image 308 ₁, and 4.) thehistogram of its intensities/frequencies is the same or closer (withrespect to a defined degree of change) to the histogram of theintensities/frequencies of reference sub-image 308 ₁. The imageharmonization component 106 can similarly apply this same harmonizationprocess to harmonize sub-image 304 ₂ with reference sub-image 308 ₂,sub-image 304 ₃ with reference sub-image 308 ₃, and sub-image 304 ₄ withreference sub-image 308 ₄. It should be appreciated that the disclosedharmonization techniques are not limited to these statistical measuresand that other statistical measures can be used.

In various embodiments, two or more statistical measures can be used incombination to increase the output of the image harmonization. Forexample, in some embodiments, the image harmonization component 106 canemploy both the mean and the standard deviation of the energy band imageintensities/frequencies. In some implementation of these embodiments,the harmonization component 106 can harmonize each energy band imagewith its corresponding reference energy band image in accordance withEquation 4 below, wherein I_(harm_i)(x) represents a harmonized (ormodified) sub-image (e.g., one of the modified sub-images 310),I_(input_i)(x) represents a sub-image for the input image 302 (e.g., oneof the sub-images 304), I_(ref_i)(x) represents a reference sub-image(e.g., one of the reference sub-images 308), and “std” is theabbreviation for standard deviation.

$\begin{matrix}{{I_{harm\_ i}(x)} = {{{mean}\left( {I_{ref\_ i}(x)} \right)} + {\frac{\left( {{I_{input\_ i}(x)} - {{{mean}\left( {I_{input\_ i}(x)} \right)}*{{std}\left( {I_{ref\_ i}(x)} \right)}}} \right.}{{std}\left( {I_{input\_ i}(x)} \right)}.}}} & {{Equation}4}\end{matrix}$

With reference again to process 300 in view of FIG. 1 , once themodified sub-images 310 have been generated at 307, the reconstructioncomponent 110 can combine the modified sub-images to reconstruct aharmonized image 312. The resulting harmonized image 312 has a greaterdegree of similarity to the reference image in terms of appearancerelative to the input image 302. For example, in the embodiment shown,the harmonized image 312 looks more like the reference image 306 thanthe input image 302 in terms of intensities.

FIG. 4 presents an example CXR image before and after imageharmonization in accordance with one or more embodiments of thedisclosed subject matter. In particular, the harmonized input image 406was generated from input image 402 using reference image 404 and theenergy band decomposition/harmonization processes of Equations 1-4. Ascan be seen in FIG. 4 , the harmonized input image 406 is more similarin appearance to the reference image 404 relative to the input image402.

Experiments were conducted to evaluate the effectiveness of the imageharmonization processes as applied to C×R image processing modelsconfigured to perform pneumoperitoneum diagnosis, segmentation, andendotracheal tube position estimation on C×R images (such as those shownin FIG. 4 ) from a specific source domain. In accordance with theseexperiments, model performance was evaluated when applied to CRX imagesfrom various other domains without harmonization (e.g. input image 402)and the same CRX images after harmonization (e.g., harmonized image 404)in accordance with the energy band decomposition/harmonization processesof Equations 1-4). The results from the different datasets consistentlyshowed that this image harmonization process increases the effectivenessof deep learning pipelines in terms of accuracy, area under the receiveroperating characteristic curve, DICE coefficient and other errormeasures related to the applications.

FIG. 5 presents an example MR scan image before and after imageharmonization in accordance with one or more embodiments of thedisclosed subject matter. In particular, the harmonized input image 504was generated from input image 502 using reference image 506 and theenergy band decomposition/harmonization processes of Equations 1-4. Ascan be seen in FIG. 5 , the harmonized input image 504 is more similarin appearance to the reference image 506 relative to the input image502.

Experiments were conducted to evaluate the effectiveness of the imageharmonization processes as applied to MR image processing modelsconfigured to perform organ and lesion segmentation on MR images (suchas those shown in FIG. 5 ) from a specific source domain. In accordancewith these experiments, model performance was evaluated when applied toMR images from various other domains without harmonization (e.g. inputimage 502) and the same MR images after harmonization (e.g., harmonizedimage 504) in accordance with the energy banddecomposition/harmonization processes of Equations 1-4). The resultsfrom the different datasets again consistently showed that this imageharmonization process increases the effectiveness of the imageprocessing models. For example, in a testing dataset of 73 organs withfull coverage, the success rate of an organ segmentation model went from86.3% to 89% after harmonization. In addition, an MR brain lesiondetection/segmentation model was tested on a dataset of 35 patients withmulti-MR sequences (T1, T1 ce, T2 and Flair). The model performanceincreased from a DICE score of 0.655+/−0.232 before harmonization to0.709+/−0.19 after harmonization.

FIG. 6 presents X-ray images of different anatomical parts before andafter image harmonization in accordance with one or more embodiments ofthe disclosed subject matter. The disclosed image harmonizationtechniques can be applied to correct various appearance variations inimages as well as to correct errors and remove artifacts. For example,FIG. 6 shows the results of the disclosed energy band imageharmonization techniques as applied to X-ray images to correct windowlevel and widow width from different X-rays images.

With reference again to FIG. 1 , in some embodiments, the decompositioncomponent 106 can employ other decomposition techniques to decompose theinput image 102 (and the reference image) into sub-images. For example,in one implementation, the decomposition component 106 can decompose theinput image 102 and the reference image by segmenting them into two ormore different regions. For instance, as applied to medical images, amedical image can be segmented into different regions (e.g. organs) ofinterest using a pre-trained segmentation model. In this regard, eachsegmented region can be considered a sub-image and will have differentfeatures (e.g., with respect to appearance features and/or spatialfeatures).

With these embodiments, the harmonization component 108 can harmonizeeach sub-image with its corresponding reference sub-image by making oneor more features of the sub-image the same or more similar (e.g., withrespect to a defined degree of change) to the corresponding features ofthe corresponding reference sub-image. For example, the harmonizationcomponent 108 can modify one or more visual features of the sub-image tobe more similar to the corresponding visual features of the referencesub-image (e.g., pixel coloration, pixel tone, saturation, etc.). Inanother example, the harmonization component 108 can modify one or morespatial features of the sub-image to be more similar to thecorresponding visual features of the reference sub-image (e.g., pixellocation, sub-image thickness, sub-image size/shape, etc.). In someimplementations, the harmonization component 108 can also calculate orotherwise generate statistical measures for a sub-image based on itsspatial and/or visual features and modify the sub-image such that itsstatistical measures are the same or similar to those of thecorresponding sub-image. For example, as applied to segmentedsub-images, the harmonization component 108 can harmonize the respectivesegmented regions for the input image 102 with corresponding segmentedregions the reference image by making one or more visual and/or spatialfeatures of the original image segments more similar to thecorresponding visual and/or spatial features of the correspondingreference image segments. The reconstruction component 110 can thenrecombine the modified sub-images to generate the harmonized image 118.

FIG. 7 illustrates an example, high-level flow diagram of acomputer-implemented process 700 for image harmonization in accordancewith one or more embodiments of the disclosed subject matter. Repetitivedescription of like elements employed in respective embodiments isomitted for sake of brevity.

At 702, method 700 comprises decomposing, by a system operativelycoupled to a processor (e.g., system 100), an input image (e.g., inputimage 302) into sub-images (e.g., sub-images 304 using decompositioncomponent 106). In various embodiments, the sub-images can compriseenergy band images with different energy frequencies generated inaccordance with equations 1-3. In other implementations, the sub-imagescan comprise different segmented regions of the input images. Variousother decomposition mechanism and resulting sub-image can also be used.At 704, method 700 comprises harmonizing (e.g., using imageharmonization component 108) the sub-images with corresponding referencesub-images (e.g., reference sub-images 308) of at least one referenceimage (e.g., reference image 306) based on two or more differentstatistical values respectively calculated for the sub-images and thecorresponding reference-sub images, resulting in transformation of thesub-images into modified sub-images images (e.g., modified sub-images310). For example, in implementations in which the sub-images are energyband images, the harmonization component 106 can modify the intensitiesof each sub-image such that one or more statistical measures of theintensities (e.g., the mean of the intensities, the standard deviationof the intensities, percentiles of the intensities, the histogram of theintensities, etc.) are the same or more similar to (e.g., with respectto a defined degree of change), to the corresponding statisticalmeasures of the corresponding reference energy band image. At 706,method 700 comprises combining (e.g., using the reconstruction component110) the modified sub-images into a harmonized image (e.g., harmonizedimage 312), the harmonized image having a more similar appearance to theat least one reference image relative to the input image.

FIG. 8 illustrates a block diagram of another example, non-limitingsystem 800 that facilitates pre-processing image harmonization for deeplearning model optimization in accordance with one or more embodimentsof the disclosed subject matter. System 800 includes same or similarcomponents as system 100 with the addition of reference image module 806and the division of image harmonization component 108 into a singleimage harmonization component 802 and a multiple reference harmonizationcomponent 804. Repetitive description of like elements employed inrespective embodiments is omitted for sake of brevity.

As noted above, in some embodiments, a same, single, pre-selectedreference image can be used to transform all input images (e.g., inputimage 102) into harmonized images for input into a particular imageprocessing model. In other embodiments, multiple (e.g., two or more)reference images can be used, or a specific reference image can beselected from amongst candidate reference images for harmonizing with aspecific input image based on the specific reference image having thegreatest degree of similarity to the specific input image. Inimplementations in which multiple references are used, the sub-imagesdecomposed from the input image 102 can be harmonized with correspondingreference sub-images of the multiple reference images using a weightingscheme for the reference sub-images determined based on degrees ofsimilarity between the input image and the respective reference images.The reference image module 806 provides various components to facilitatethis end, including vectorization component 808, clustering component810, reference set generation component 812, similarity scoringcomponent 814 and reference image selection component 816.

In some embodiments, the vectorization component 808 can generatefeature vectors for the input image 102 and candidate reference imagesincluded in the reference image data source 114. The feature vectors canbe used to determine a degree of similarity between the input image 102and the respective candidate reference images. In some implementations,the feature vectors can be generated based on one or more of thestatistical measures that will be used to harmonize the sub-images withthe corresponding reference sub-images. For example, in implementationsin which the images are decomposed into energy band images, thevectorization component 808 can generate the feature vectors based onone or more statistics of their respective energy band images (e.g., themeans of the energy band image intensity values for each energy-bandimage, the standard deviations of the energy band image intensity valuesfor each energy-band image, the percentiles of the energy band imageintensity values for each energy-band image, etc.). In some embodiments,the vectorization component 808 can also employ other image-basedfeatures (e.g., dose usage, exposure setting, photon receivingmaterials, FOV, demography, contrast vs. non-contrast, etc.) and/ornon-image based features (e.g., patient specific features such aspatient demographics, patient medical history, etc.) included inmetadata associated with the respective images to generate the featurevectors.

Additionally, or alternatively, the vectorization component 808 cangenerate the feature vectors based on the outputs of one or more fullyconnected layers of one or more pretrained deep neural network modelstrained on a corpus of similar images (e.g., using an ImageNet datasetor the like). For example, the pretrained deep neural network caninclude a classic neural network model, a convolutional neural networkmodel, a recurrent neural network model, an autoencoder network, or thelike. The task of the pretrained model can vary, so long as it isconfigured to consistently identify and extract (as the output of one ormore fully connected layers) relevant and distinguishing features fromthe input images. For example, in one implementation, the pretrainedmodel can comprise a deep autoencoder model than consists of an encoderneural network and a decoder neural network. Such autoencoder modelsgenerally work by mapping input data into a latent representation infeature space which is then decoded by the decoder network toreconstruct the input data from the reduced feature representation inthe latent space. As applied to the disclosed subject matter, theautoencoder model can be trained on corpus of images similar to theinput image (e.g., the same type, such as C×R images) yet from variousdomains or that otherwise have appearance variations. The trainedencoder network of the autoencoder can be applied to the input image andthe reference images to extract reduced feature sets for the respectiveimages which can be used to generate feature vectors for the respectiveimages.

As noted above, the reference image or images used for harmonization canbe or correspond to the training images used to train a particular imageprocessing model. In some embodiments, the vectorization component 808can generate feature vectors for each of the training images or arandomly selected reduced subset of the training images. These trainingimages can thus be considered candidate reference images in thiscontext. In some embodiments, the similarity scoring component 814 cancompare the feature vector for the input image 102 to the featurevectors for all (or the reduced subset) of the candidate referenceimages and determine a degree of similarity between the input imagefeature vector and the feature vectors of the respective candidatereference images. For example, the similarity scoring component 814 cangenerate a similarity score for each input image/candidate referenceimage pair based on a degree of similarity between their featurevectors.

In some implementations of these embodiments, the reference imageselection component 816 can select a single candidate reference image tobe used for harmonization with the input image 102 based on thesimilarity score for the input image/candidate reference image pairbeing the highest (i.e., the most similar or best match). With theseimplementations, the image harmonization component 108 can employ thesingle reference harmonization component 802 to harmonize the referenceimage with the input image using the technique described above.

In another implementation, the reference set generation component 816can select the top N highest (wherein N is a defined integer greaterthan one) scoring candidate reference images for inclusion in areference image set be used for harmonization with the input image 102in accordance with the multiple reference harmonization scheme. Inanother implementation in which multiple reference images are used, toprovide a more diverse representation of the training datasetpopulation, the reference set generation component 812 can select asubset of the candidate reference images that have varying degrees ofsimilarity with the input image (e.g., that have varying similarityscores) for inclusion in the reference image set. For example, thereference set generation component 812 can select the top scoringcandidate reference image (e.g., having the closest degree of similarityto the input image), the lowest scoring candidate reference image, and amiddle scoring candidate reference image.

Additionally, or alternatively, rather than comparing the input imagefeature vector with the feature vectors for all of the training images(i.e., the candidate reference images), the reference set generationcomponent 812 can generate a fixed reference image set of referenceimages that provide a diverse representation of images included in thetraining dataset. This fixed set of reference images can then be appliedto for each input image 102. According to these embodiments, theclustering component 810 can cluster the candidate reference images intodifferent groups of related images based on similarities and differencesbetween their feature vectors (e.g., using a suitable clusteringalgorithm such as k-means clustering or the like). The number ofclusters can vary based on the desired number of reference images forinclusion in the reference image set. The reference set generationcomponent 812 can further select one or more representative referenceimages from each cluster for inclusion in the fixed reference image set.In this regard, the reference set generation component 812 can pick afew representative reference images with vectors that span the trainingdataset population as much as possible.

In some implementations of these embodiments, after the fixed referenceimage set is generated, each time a new input image is received forharmonization, the vectorization component 808 can generate a featurevector for the input image. The similarity scoring component 814 canfurther generate similarity scores for each input image/reference imageincluded in the set based on their respective feature vectors. Thereference image selection component 816 can further select one of thereference images included in the set for harmonization with the inputimage using the single reference harmonization component 802.

In various alternative embodiments in which multiple images are selected(e.g., the images included in a reference image set), the multiplereference harmonization component 804 can harmonize the input image 102using each of the reference images in the set and a weighting schemebased on the degree of similarity of the input image feature vector witheach of the feature vectors of the reference images included in the set.For example, assume the reference image set has 10 reference images,each with different feature vectors. At runtime, the similarity scoringcomponent 814 can generate a similarity score for each of the 10reference images that reflects a degree of similarity between therespective feature vectors of the reference images and the input imagefeature vector. The multiple reference harmonization component 804 canfurther harmonize each sub-image of the input image the correspondingreference sub-images using a weighting scheme for reference sub-imagesdetermined based on the similarity scores. For example, each of the 10reference images will be decomposed into sub-images, resulting in groupsof 10 sub-images corresponding to a same filter pass (e.g., 10 firstreference sub-images, 10 second pass reference sub-images, etc.). Themultiple reference harmonization component 804 can then harmonize thefirst pass sub-image for the input image with each of the 10corresponding reference sub-images, using a weighting scheme forreference sub-images that give greater weight to sub-images associatedwith higher (e.g., more similar) similarity scores. The multiplereference harmonization component 804 can repeat this harmonizationprocess for each sub-image and the corresponding group of referencesub-images to generate the modified sub-images.

In one implementation as applied to extend Equation 4 to multiplereference image harmonization, Equation 4 can transformed into Equation5, wherein sm(I(x), Ir^(r)(x)) is the normalized similarity scorebetween an input image and the r^(th) reference image in the set. Thesimilarity score can be the absolute correlation between the featurevector of the input image and the feature vector of the r^(th) referenceimage.

$\begin{matrix}{{I_{harm\_ i}(x)} = {\sum\limits_{r}{{{sm}\left( {{I(x)},{{Ir}^{r}(x)}} \right)}*{\left( {{{mean}\left( \text{⁠}{I_{ref\_ i}\left( \text{⁠}x \right)} \right)} + \text{⁠}\frac{\left( {{I_{input\_ i}(x)} - {{{mean}\left( {I_{input\_ i}(x)} \right)}*{{std}\left( {I_{ref\_ i}(x)} \right)}}} \right.}{{std}\left( {I_{input\_ i}(x)} \right)}}\text{⁠} \right)\text{⁠}.}}}} & {{Equation}5}\end{matrix}$

In Alternatively, in accordance with Equation 5, among the multiplereferences, the multiple reference harmonization component 804 can applya similarity metric of 1.0 to the most similar vector and 0 to allothers, resulting in harmonization of the input image to only theclosest matching reference image.

FIG. 9 illustrates an example, high-level flow diagram of anothercomputer-implemented process 900 for image harmonization in accordancewith one or more embodiments of the disclosed subject matter. Repetitivedescription of like elements employed in respective embodiments isomitted for sake of brevity.

At 902, method 900 comprises decomposing, by a system operativelycoupled to a processor (e.g., system 800), an input image (e.g., inputimage 102, input image 302 or the like) into sub-images (e.g.,sub-images 304 using decomposition component 106). At 904, method 900comprises selecting, by the system (e.g., using reference imageselection component 816), a reference image (e.g., reference image 306)from a set of reference images based on a degree of similarity between afirst feature vector for the input image and a second feature vector forthe reference image. At 906, method 900 comprises harmonizing (e.g.,using single reference harmonization component 802) the sub-images withcorresponding reference sub-images (e.g., reference sub-images 308) ofthe reference image, resulting in transformation of the sub-images intomodified sub-images images (e.g., modified sub-images 310). At 908,method 900 comprises combining (e.g., using the reconstruction component110) the modified sub-images into a harmonized image (e.g., harmonizedimage 312), the harmonized image having a more similar appearance to thereference image relative to the input image.

FIG. 10 illustrates an example, high-level flow diagram of anothercomputer-implemented process 1000 for image harmonization in accordancewith one or more embodiments of the disclosed subject matter. Repetitivedescription of like elements employed in respective embodiments isomitted for sake of brevity.

At 1002, method 1000 comprises decomposing, by a system operativelycoupled to a processor (e.g., system 800), an input image (e.g., inputimage 102, input image 302 or the like) into sub-images (e.g.,sub-images 304 using decomposition component 106). At 1004, method 900comprises harmonizing, by the system, the sub-images with referencesub-images of respective reference images (e.g., using multiplereference harmonization component 804), resulting in transformation ofthe sub-images into modified sub-images, wherein the harmonizingcomprises using a weighting scheme for the reference sub-imagesdetermined based on degrees of similarity between the input image andthe respective reference images. At 1008, method 1000 comprisescombining (e.g., using the reconstruction component 110) the modifiedsub-images into a harmonized image, the harmonized image having a moresimilar appearance to the respective reference images relative to theinput image.

FIG. 11 illustrates a block diagram of another example, non-limitingsystem 1100 that facilitates pre-processing image harmonization for deeplearning model optimization in accordance with one or more embodimentsof the disclosed subject matter. System 1100 includes same or similarcomponents as system 800 with the addition of training component 1102,harmonization training images 1104, one or more harmonization models,and harmonization model application component 1108. Repetitivedescription of like elements employed in respective embodiments isomitted for sake of brevity.

System 1100 provides a deep learning extension to the above describedimage harmonization techniques described above. In particular, system1100 uses harmonized images and/or modified sub-images generated inaccordance with the harmonization techniques described above asground-truth training samples for training one or more ML/AI models totransform input images with appearance variations into harmonized images(e.g., harmonized with an image processing model's training dataset)prior to input into the image processing model. Such ML/AI models arereferred to herein as image harmonization models 1106. For example, theone or more image harmonization models 1106 can comprise one or moredeep learning models, such an autoencoder, a generative adversarialautoencoder, a generative adversarial network (GAN), or the like. Withthese embodiments, once trained, the one or more image harmonizationmodels 1102 can be applied to new input images (not shown) by theharmonization model application component 1108 to transform new inputimages with different feature variations into harmonized images thathave a more similar appearance to the original source training images.

In this regard, system 1100 can include a plurality of harmonizationtraining images 1104 that can include a diverse set of variationalimages with feature variations relative to the original source domainimages used to train/develop a particular image processing model. Forexample, the harmonization training images 1104 can include a pluralityof images of the same type as the source domain images (e.g., both beingC×R images) yet from one or more different domains relative to thesource images. At least some of these diverse variational harmonizationimages 1104 can processed using the image harmonization techniquesdescribed above to generate harmonized images for the respectiveharmonization training images. For example, all (or some) of theharmonization training image 1104 (or at least some) can be decomposedinto sub-images by the decomposition component 106, harmonized withcorresponding reference sub-images by the image harmonization component108 and recombined into harmonized images (such as harmonized image 118)by the reconstruction component 110 in accordance with the techniquesdescribed above. These harmonization training image/harmonized imagepairs and/or sub-image/modified sub-image pairs can also be stored withthe harmonization training images 1104 and the training component 1102can use them as the ground-truth training samples for training one orharmonization models 1106. In some implementations, the harmonizationtraining images 1104 can also include at least some images that withoutground-truth samples (e.g., that have not been processed to generateground-truth harmonized images).

The disclosed techniques for developing and applying one or more imageharmonization models 1106 provide a highly efficient end-to-end processfor image harmonization as the mechanism for generating the ground-truthtraining data is automated. In this regard, one major limitation inML/AI model development is the availability of ground-truth trainingdata, which most often requires manual annotation, a tedious and costlyendeavor. With the disclosed techniques, the ground-truth training datacan be generated without any manual intervention, thus providing forlimitless, low-cost, and efficient ground-truth training datageneration. In addition, the usage of a trained image harmonizationmodel to transform input images into harmonized images reduces theoverall processing time for image harmonization relative to abovedescribed decomposition, sub-image harmonization and reconstructionprocess described above.

The harmonization model-based image harmonization techniques include twodifferent approaches; a full image approach and a sub-image approach.The full image approach is described in greater detail with reference toFIG. 12 and the sub-image approach is described in greater detail withreference to FIGS. 13A and 13B.

In this regard, FIG. 12 illustrates training and usage of an imagefull-image harmonization model 1204 in accordance with the full imageapproach. In particular, process 1200 illustrates model training andprocess 1201 illustrates application of the trained model to a new inputimage (e.g., input image 1210) to generate a model generated (MG)harmonized image 1212. With reference to process 1200, in accordancewith the full image approach, the training images 1202 can include aplurality of variational images (e.g., with appearance variationsrelative to source domain images used to train a particular imageprocessing model). At least some of the training images 1202 haveground-truth harmonized images generated therefore using the imageharmonization techniques described above. In this regard, each of theground-truth harmonized image 1206 is a harmonized image that wasgenerated for a corresponding training image 1202 using thedecomposition component 106, the image harmonization component 108 andthe reconstruction component 110. In accordance with this full imageapproach, the sub-images and corresponding modified sub-images generatedfor each of the training images 1202 during the image harmonizationprocess can be discarded.

The type of the image full-image harmonization model 1204 can vary andthus the training process employed by the training component 1102 totrain the image full-image harmonization model 1204 can also vary. At ahigh level, the training process involves training the full-imageharmonization model 1204 to transform the training image 1202 withground-truth harmonized images into their ground-truth harmonizedimages. More particularly, the training process involves applying thefull-image harmonization model 1204 to the training images 1202 togenerate model generated harmonized images 1206. The ground-truthharmonized training images 1206 are further compared to the modelgenerated harmonized images 1208 and the full-image harmonization model1204 parameters are then tuned based on the differences between theground-truth harmonized images 1206 and the model generated harmonizedimages 1208. For example, in one implementation, the full-imageharmonization model 1204 can comprises a generative model followed by adiscriminator and the training can involve unsupervised and/orsemi-supervised training.

In the embodiment shown, an asterisk (e.g., harmonization model 204′) isused to indicate the harmonization model is being trained. The asteriskis removed in process 1201 to indicate the model training is complete.In this regard, after the full-image harmonization model 1204 has betrained, the harmonization model application component 1108 can applythe harmonization model to new input images (e.g., input image 1210) totransform it into a model generated harmonized image 1212 having agreater degree of similarity in appearance to the source domain imagesrelative to the input image 1210.

FIG. 13A illustrates an example process 1300 for training sub-imageharmonization models 1304 _(1-i) in accordance with the sub-imagemodel-based harmonization approach. In accordance with the sub-imageapproach, a separate sub-image harmonization model can be trained foreach group of related sub-images to transform them into modifiedsub-images. With this embodiment, the groups of sub-images andcorresponding modified sub-images generated for each of the trainingimages 1202 during the image harmonization process are used as theground-truth training samples.

For example, as shown in process 1300, each of the training images 1202can be decomposed into different sub-images (e.g., different energy bandimages) in accordance with the decomposition techniques described withreference to the decomposition component 106. For example, a singletraining image can be decomposed into a first training sub-image, asecond training sub-image and so on up until the i^(th) sub-image(wherein i=1, 2 . . . , B). In the embodiment shown, the sub-images of asame type can be grouped together as separate groups (e.g., the firstpass sub-images grouped together, the second pass-sub images groupedtogether, and so on). For example, in the embodiment shown, thedifferent groups of sub-images are respectively identified as firsttraining sub-images 1301 ₁, second training sub-images 1302 ₂, and so onup to the i^(th) group. At least some of the sub-images in each trainingcan be processed using the image harmonization techniques of the imageharmonization component 108 to generate modified sub-images for thecorresponding sub-images. For example, in the embodiment shown, eachgroup of sub-images has at least some ground-truth (GT) modifiedsub-images (e.g., GT first modified sub-images 1308 ₁, GT secondmodified sub-images 1308 ₂ and so on). These GT modified sub-images canbe used as the ground-truth for training separate sub-imageharmonization models 1304 to transform the sub-images of a particulargroup into model generated modified sub-images 1306. For example, afirst sub-image harmonization model 1304 ₁, can be trained to transformthe first training sub-images 1302 ₁ into MG-first modified sub-images1306 ₁ using the GT first modified sub-images 1308 ₁ as theground-truth. A second sub-image harmonization model 1304 ₂, can betrained to transform the second training sub-images 1302 ₂ intoMG-second modified sub-images 1306 ₂ using the GT second modifiedsub-images 1308 ₂ as the ground-truth. This process can be repeated foreach of the sub-image groups up to the i^(th) group.

The type of the sub-image image full-image harmonization model 1204 canvary and thus the training process employed by the training component1102 to train the image sub-image harmonization model 1304 can alsovary. At a high level, the training process can involve applying thesub-harmonization model 1304 _(i) to its training sub-images 1302 _(i)to generate model generated harmonized modified sub-images 1306 _(i).The ground-truth modified training sub-images 1308 _(i) are furthercompared to the model output and the sub-image harmonization model 1304_(i) parameters are then tuned based on the differences between theground-truth modified sub-images 1308 _(i) and the model generatedmodified sub-images 1306 _(i). For example, in one implementation, thesub-image harmonization models 1304 can comprise generative modelsfollowed by a discriminator and the training can involve unsupervisedand/or semi-supervised training.

With reference now to FIG. 13B, process 1301 illustrates application ofthe (trained) sub-image harmonization models 1304 for pre-processingimage harmonization in accordance with one or more embodiments of thedisclosed subject matter. In the embodiment shown in FIG. 13A, anasterisk (e.g., sub-image harmonization model 1304′) is used to indicatethe sub-image harmonization model is being trained. The asterisk isremoved in process 1301 to indicate the model training is complete. Inthis regard, after the sub-image harmonization models 1304 _(1-i), havebe trained, the decomposition component 106 can decompose a new inputimage 1208 into sub images, respectively identified in process 1301 assub-images 1310 _(1-i). The harmonization model application component1108 can further apply the respective sub-image harmonization models1304 to their appropriate sub-images 1310 _(1-i) to transform therespective sub-images 1310 _(1-i) into model generated modifiedsub-images 1312 _(1-i). At 1304, the reconstruction component 110 canthen combine the model generated modified sub-images 1312 _(1-i) toreconstruct a model generated harmonized image 1316 having a greaterdegree of similarity in appearance to the source domain images relativeto the input image 1210.

FIG. 14 illustrates an example, high-level flow diagram of acomputer-implemented process 1400 for image harmonization using afull-image harmonization model in accordance with one or moreembodiments of the disclosed subject matter. Repetitive description oflike elements employed in respective embodiments is omitted for sake ofbrevity.

At 1402, method 1400 comprises generating, by a system operativelycoupled to a processor (e.g., system 1100), ground-truth harmonizedimages (e.g., ground-truth harmonized images 1206) for respectivetraining images (e.g., training images 1202) with different appearancevariations (e.g., using single reference harmonization component 802 ormultiple reference harmonization component 804), wherein the generatingcomprises employing at least one reference image and wherein theground-truth harmonized images have a more similar appearance to the atleast one reference image relative to the respective training images.Additional details regarding an example process for generating theground-truth harmonized images are described below with reference toprocess 1500 and operations 1502-1506. At 1404, method 1400 furthercomprises training, by the system (e.g., using training component 1102),a full-image harmonization model (e.g., full-image harmonization model1204) to transform the respective training images into the ground-truthharmonized images. At 1406, method 1400 further comprises applying, bythe system (e.g., using harmonization model application component 1108),the full-image harmonization model to a new input image (e.g., inputimage 1210) to transform the new input image into a new harmonized image(e.g., model-generated harmonized image 1210) that has a more similarappearance to the at least one reference image relative to the new inputimage.

FIG. 15 illustrates an example, high-level flow diagram of acomputer-implemented process 1500 for developing a full-imageharmonization model in accordance with one or more embodiments of thedisclosed subject matter. Repetitive description of like elementsemployed in respective embodiments is omitted for sake of brevity.

At 1502, method 1500 comprises decomposing (e.g., using decompositioncomponent 106), by a system operatively coupled to a processor (e.g.,system 1100), training images (e.g., training images 1202) withdifferent appearance variations into sub-images (e.g., trainingsub-images 1302 _(1-i)). At 1504, method 1500 comprises harmonizing, bythe system, the sub-images with corresponding reference sub-images ofreference images resulting in transformation of the sub-images intomodified sub-images (e.g., the ground-truth modified sub-images 1308_(1-i)), using the techniques described above with reference to theimage harmonization component 108 (e.g., using single referenceharmonization component 802 or multiple reference harmonizationcomponent 804). At 1506, method 1500 further comprises combining, by thesystem, groups of the modified sub-images associated with respectivetraining images of the training images to generate ground-truthharmonized images (e.g., ground-truth harmonized images 1206) for therespective training images (e.g., using reconstruction component 110).For example, as described with reference to FIGS. 3 and 11 , eachtraining image is decomposed into a group of sub-images (e.g.,sub-images 304) which are then transformed into modified sub-images(e.g., modified sub-images 310) using corresponding reference sub-images(e.g., reference sub-images 308) and the image harmonization techniquesdescribed with reference to the image harmonization component 108. Thegroup of modified sub-images (e.g., modified sub-images 310) are thencombined by the reconstruction component 110 to generate a ground-truthharmonized image (e.g., harmonized image 312) for the training image. At1508, method 1500 further comprises training, by the system (e.g., usingtraining component 1102), a full-image harmonization model (e.g.,full-image harmonization model 1204) to transform the training imagesinto the ground-truth harmonized images, as described with reference toFIG. 12 and process 1200.

FIG. 16 illustrates an example, high-level flow diagram of acomputer-implemented process 1600 for image harmonization usingsub-image harmonization models in accordance with one or moreembodiments of the disclosed subject matter. Repetitive description oflike elements employed in respective embodiments is omitted for sake ofbrevity.

At 1602, method 6100 comprises generating, by a system operativelycoupled to a processor (e.g., system 1100), ground-truth sub-images(e.g., ground-truth modified sub-images 1308 _(1-i)) for respectivesub-images (e.g., training sub-images 1302 _(1-i)) decomposed fromtraining images (e.g., training images 1202), wherein the generatingcomprise employing at least one reference image. For example, thegeneration of the modified sub-images can be performed by thedecomposition component 106 and image harmonization component 108 inaccordance with operations 1502-1504 of process 1500, (e.g., byharmonizing the sub-images with corresponding reference sub-images,resulting in a transformation of the sub-images into the ground-truthsub-images). At 1604, method 1600 further comprises training, by thesystem, sub-image harmonization models (e.g., sub-image harmonizationmodels 1304 _(1-i)) to transform the sub-images into the ground-truthsub-images (e.g., using training component 1102), as described withreference to FIG. 13A and process 1300.

At 1608, method 1600 further comprises applying, by the system (e.g.,using the harmonization model application component 1108), the sub-imagemachine learning models to transform new sub-images (e.g., sub-images1310 _(1-i)) decomposed from an input image (e.g., input image 1210)into new modified sub-images (e.g., model-generated modified sub-images1312 _(1-i)), as described with reference to FIG. 13B and process 1301.At 1610, method 1600 comprises combining, by the system (e.g., usingreconstruction component 110), the new modified sub-images to generate aharmonized image (model-generated harmonized image 1316) for the inputimage, the harmonized image having a more similar appearance to the atleast one reference images relative to the input image.

EXAMPLE OPERATING ENVIRONMENT

One or more embodiments can be a system, a method, and/or a computerprogram product at any possible technical detail level of integration.The computer program product can include a computer readable storagemedium (or media) having computer readable program instructions thereonfor causing a processor to carry out aspects of the present invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium can be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network can comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention can be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, configuration data for integrated circuitry, oreither source code or object code written in any combination of one ormore programming languages, including an object oriented programminglanguage such as Smalltalk, C++, or the like, and procedural programminglanguages, such as the “C” programming language or similar programminglanguages. The computer readable program instructions can executeentirely on the user's computer, partly on the user's computer, as astand-alone software package, partly on the user's computer and partlyon a remote computer or entirely on the remote computer or server. Inthe latter scenario, the remote computer can be connected to the user'scomputer through any type of network, including a local area network(LAN) or a wide area network (WAN), or the connection can be made to anexternal computer (for example, through the Internet using an InternetService Provider). In some embodiments, electronic circuitry including,for example, programmable logic circuitry, field-programmable gatearrays (FPGA), or programmable logic arrays (PLA) can execute thecomputer readable program instructions by utilizing state information ofthe computer readable program instructions to personalize the electroniccircuitry, in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It can be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions can be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionscan also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions can also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational steps to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams can represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the blocks can occur out of theorder noted in the Figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks cansometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions.

In connection with FIG. 17 , the systems and processes described belowcan be embodied within hardware, such as a single integrated circuit(IC) chip, multiple ICs, an application specific integrated circuit(ASIC), or the like. Further, the order in which some or all of theprocess blocks appear in each process should not be deemed limiting.Rather, it should be understood that some of the process blocks can beexecuted in a variety of orders, not all of which can be explicitlyillustrated herein.

With reference to FIG. 17 , an example environment 1700 for implementingvarious aspects of the claimed subject matter includes a computer 1702.The computer 1702 includes a processing unit 1704, a system memory 1706,a codec 1735, and a system bus 1708. The system bus 1708 couples systemcomponents including, but not limited to, the system memory 1706 to theprocessing unit 1704. The processing unit 1704 can be any of variousavailable processors. Dual microprocessors and other multiprocessorarchitectures also can be employed as the processing unit 1704.

The system bus 1708 can be any of several types of bus structure(s)including the memory bus or memory controller, a peripheral bus orexternal bus, or a local bus using any variety of available busarchitectures including, but not limited to, Industrial StandardArchitecture (ISA), Micro-Channel Architecture (MSA), Extended ISA(EISA), Intelligent Drive Electronics (IDE), VESA Local Bus (VLB),Peripheral Component Interconnect (PCI), Card Bus, Universal Serial Bus(USB), Advanced Graphics Port (AGP), Personal Computer Memory CardInternational Association bus (PCMCIA), Firewire (IEEE 1394), and SmallComputer Systems Interface (SCSI).

The system memory 1706 includes volatile memory 1710 and non-volatilememory 1712, which can employ one or more of the disclosed memoryarchitectures, in various embodiments. The basic input/output system(BIOS), containing the basic routines to transfer information betweenelements within the computer 1702, such as during start-up, is stored innon-volatile memory 1712. In addition, according to present innovations,codec 1735 can include at least one of an encoder or decoder, whereinthe at least one of an encoder or decoder can consist of hardware,software, or a combination of hardware and software. Although, codec1735 is depicted as a separate component, codec 1735 can be containedwithin non-volatile memory 1712. By way of illustration, and notlimitation, non-volatile memory 1712 can include read only memory (ROM),programmable ROM (PROM), electrically programmable ROM (EPROM),electrically erasable programmable ROM (EEPROM), Flash memory, 3D Flashmemory, or resistive memory such as resistive random access memory(RRAM). Non-volatile memory 1712 can employ one or more of the disclosedmemory devices, in at least some embodiments. Moreover, non-volatilememory 1712 can be computer memory (e.g., physically integrated withcomputer 1702 or a mainboard thereof), or removable memory. Examples ofsuitable removable memory with which disclosed embodiments can beimplemented can include a secure digital (SD) card, a compact Flash (CF)card, a universal serial bus (USB) memory stick, or the like. Volatilememory 1710 includes random access memory (RAM), which acts as externalcache memory, and can also employ one or more disclosed memory devicesin various embodiments. By way of illustration and not limitation, RAMis available in many forms such as static RAM (SRAM), dynamic RAM(DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM),and enhanced SDRAM (ESDRAM) and so forth.

Computer 1702 can also include removable/non-removable,volatile/non-volatile computer storage medium. FIG. 17 illustrates, forexample, disk storage 1714. Disk storage 1714 includes, but is notlimited to, devices like a magnetic disk drive, solid state disk (SSD),flash memory card, or memory stick. In addition, disk storage 1714 caninclude storage medium separately or in combination with other storagemedium including, but not limited to, an optical disk drive such as acompact disk ROM device (CD-ROM), CD recordable drive (CD-R Drive), CDrewritable drive (CD-RW Drive) or a digital versatile disk ROM drive(DVD-ROM). To facilitate connection of the disk storage 1714 to thesystem bus 1708, a removable or non-removable interface is typicallyused, such as interface 1716. It is appreciated that disk storage 1714can store information related to a user. Such information might bestored at or provided to a server or to an application running on a userdevice. In one embodiment, the user can be notified (e.g., by way ofoutput device(s) 1736) of the types of information that are stored todisk storage 1714 or transmitted to the server or application. The usercan be provided the opportunity to opt-in or opt-out of having suchinformation collected or shared with the server or application (e.g., byway of input from input device(s) 1728).

It is to be appreciated that FIG. 17 describes software that acts as anintermediary between users and the basic computer resources described inthe suitable operating environment 1700. Such software includes anoperating system 1718. Operating system 1718, which can be stored ondisk storage 1714, acts to control and allocate resources of thecomputer 1702. Applications 1720 take advantage of the management ofresources by operating system 1718 through program modules 1724, andprogram data 1726, such as the boot/shutdown transaction table and thelike, stored either in system memory 1706 or on disk storage 1714. It isto be appreciated that the claimed subject matter can be implementedwith various operating systems or combinations of operating systems.

A user enters commands or information into the computer 1702 throughinput device(s) 1728. Input devices 1728 include, but are not limitedto, a pointing device such as a mouse, trackball, stylus, touch pad,keyboard, microphone, joystick, game pad, satellite dish, scanner, TVtuner card, digital camera, digital video camera, web camera, and thelike. These and other input devices connect to the processing unit 1704through the system bus 1708 via interface port(s) 1730. Interfaceport(s) 1730 include, for example, a serial port, a parallel port, agame port, and a universal serial bus (USB). Output device(s) 1736 usesome of the same type of ports as input device(s) 1728. Thus, forexample, a USB port can be used to provide input to computer 1702 and tooutput information from computer 1702 to an output device 1736. Outputadapter 1734 is provided to illustrate that there are some outputdevices 1736 like monitors, speakers, and printers, among other outputdevices 1736, which require special adapters. The output adapters 1734include, by way of illustration and not limitation, video and soundcards that provide a means of connection between the output device 1736and the system bus 1708. It should be noted that other devices orsystems of devices provide both input and output capabilities such asremote computer(s) 1738.

Computer 1702 can operate in a networked environment using logicalconnections to one or more remote computers, such as remote computer(s)1738. The remote computer(s) 1738 can be a personal computer, a server,a router, a network PC, a workstation, a microprocessor based appliance,a peer device, a smart phone, a tablet, or other network node, andtypically includes many of the elements described relative to computer1702. For purposes of brevity, only a memory storage device 1740 isillustrated with remote computer(s) 1738. Remote computer(s) 1738 islogically connected to computer 1702 through a network interface 1742and then connected via communication connection(s) 1744. Networkinterface 1742 encompasses wire or wireless communication networks suchas local-area networks (LAN) and wide-area networks (WAN) and cellularnetworks. LAN technologies include Fiber Distributed Data Interface(FDDI), Copper Distributed Data Interface (CDDI), Ethernet, Token Ringand the like. WAN technologies include, but are not limited to,point-to-point links, circuit switching networks like IntegratedServices Digital Networks (ISDN) and variations thereon, packetswitching networks, and Digital Subscriber Lines (DSL).

Communication connection(s) 1744 refers to the hardware/softwareemployed to connect the network interface 1742 to the bus 1708. Whilecommunication connection 1744 is shown for illustrative clarity insidecomputer 1702, it can also be external to computer 1702. Thehardware/software necessary for connection to the network interface 1742includes, for exemplary purposes only, internal and externaltechnologies such as, modems including regular telephone grade modems,cable modems and DSL modems, ISDN adapters, and wired and wirelessEthernet cards, hubs, and routers.

While the subject matter has been described above in the general contextof computer-executable instructions of a computer program product thatruns on a computer and/or computers, those skilled in the art willrecognize that this disclosure also can or can be implemented incombination with other program modules. Generally, program modulesinclude routines, programs, components, data structures, etc. thatperform particular tasks and/or implement particular abstract datatypes. Moreover, those skilled in the art will appreciate that theinventive computer-implemented methods can be practiced with othercomputer system configurations, including single-processor ormultiprocessor computer systems, mini-computing devices, mainframecomputers, as well as computers, hand-held computing devices (e.g., PDA,phone), microprocessor-based or programmable consumer or industrialelectronics, and the like. The illustrated aspects can also be practicedin distributed computing environments where tasks are performed byremote processing devices that are linked through a communicationsnetwork. However, some, if not all aspects of this disclosure can bepracticed on stand-alone computers. In a distributed computingenvironment, program modules can be located in both local and remotememory storage devices.

As used in this application, the terms “component,” “system,”“platform,” “interface,” and the like, can refer to and/or can include acomputer-related entity or an entity related to an operational machinewith one or more specific functionalities. The entities disclosed hereincan be either hardware, a combination of hardware and software,software, or software in execution. For example, a component can be, butis not limited to being, a process running on a processor, a processor,an object, an executable, a thread of execution, a program, and/or acomputer. By way of illustration, both an application running on aserver and the server can be a component. One or more components canreside within a process and/or thread of execution and a component canbe localized on one computer and/or distributed between two or morecomputers. In another example, respective components can execute fromvarious computer readable media having various data structures storedthereon. The components can communicate via local and/or remoteprocesses such as in accordance with a signal having one or more datapackets (e.g., data from one component interacting with anothercomponent in a local system, distributed system, and/or across a networksuch as the Internet with other systems via the signal). As anotherexample, a component can be an apparatus with specific functionalityprovided by mechanical parts operated by electric or electroniccircuitry, which is operated by a software or firmware applicationexecuted by a processor. In such a case, the processor can be internalor external to the apparatus and can execute at least a part of thesoftware or firmware application. As yet another example, a componentcan be an apparatus that provides specific functionality throughelectronic components without mechanical parts, wherein the electroniccomponents can include a processor or other means to execute software orfirmware that confers at least in part the functionality of theelectronic components. In an aspect, a component can emulate anelectronic component via a virtual machine, e.g., within a cloudcomputing system.

In addition, the term “or” is intended to mean an inclusive “or” ratherthan an exclusive “or.” That is, unless specified otherwise, or clearfrom context, “X employs A or B” is intended to mean any of the naturalinclusive permutations. That is, if X employs A; X employs B; or Xemploys both A and B, then “X employs A or B” is satisfied under any ofthe foregoing instances. Moreover, articles “a” and “an” as used in thesubject specification and annexed drawings should generally be construedto mean “one or more” unless specified otherwise or clear from contextto be directed to a singular form. As used herein, the terms “example”and/or “exemplary” are utilized to mean serving as an example, instance,or illustration and are intended to be non-limiting. For the avoidanceof doubt, the subject matter disclosed herein is not limited by suchexamples. In addition, any aspect or design described herein as an“example” and/or “exemplary” is not necessarily to be construed aspreferred or advantageous over other aspects or designs, nor is it meantto preclude equivalent exemplary structures and techniques known tothose of ordinary skill in the art.

As it is employed in the subject specification, the term “processor” canrefer to substantially any computing processing unit or devicecomprising, but not limited to, single-core processors;single-processors with software multithread execution capability;multi-core processors; multi-core processors with software multithreadexecution capability; multi-core processors with hardware multithreadtechnology; parallel platforms; and parallel platforms with distributedshared memory. Additionally, a processor can refer to an integratedcircuit, an application specific integrated circuit (ASIC), a digitalsignal processor (DSP), a field programmable gate array (FPGA), aprogrammable logic controller (PLC), a complex programmable logic device(CPLD), a discrete gate or transistor logic, discrete hardwarecomponents, or any combination thereof designed to perform the functionsdescribed herein. Further, processors can exploit nano-scalearchitectures such as, but not limited to, molecular and quantum-dotbased transistors, switches and gates, in order to optimize space usageor enhance performance of user equipment. A processor can also beimplemented as a combination of computing processing units. In thisdisclosure, terms such as “store,” “storage,” “data store,” datastorage,” “database,” and substantially any other information storagecomponent relevant to operation and functionality of a component areutilized to refer to “memory components,” entities embodied in a“memory,” or components comprising a memory. It is to be appreciatedthat memory and/or memory components described herein can be eithervolatile memory or nonvolatile memory, or can include both volatile andnonvolatile memory. By way of illustration, and not limitation,nonvolatile memory can include read only memory (ROM), programmable ROM(PROM), electrically programmable ROM (EPROM), electrically erasable ROM(EEPROM), flash memory, or nonvolatile random access memory (RAM) (e.g.,ferroelectric RAM (FeRAM). Volatile memory can include RAM, which canact as external cache memory, for example. By way of illustration andnot limitation, RAM is available in many forms such as synchronous RAM(SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rateSDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM),direct Rambus RAM (DRRAM), direct Rambus dynamic RAM (DRDRAM), andRambus dynamic RAM (RDRAM). Additionally, the disclosed memorycomponents of systems or computer-implemented methods herein areintended to include, without being limited to including, these and anyother suitable types of memory.

What has been described above include mere examples of systems andcomputer-implemented methods. It is, of course, not possible to describeevery conceivable combination of components or computer-implementedmethods for purposes of describing this disclosure, but one of ordinaryskill in the art can recognize that many further combinations andpermutations of this disclosure are possible. Furthermore, to the extentthat the terms “includes,” “has,” “possesses,” and the like are used inthe detailed description, claims, appendices and drawings such terms areintended to be inclusive in a manner similar to the term “comprising” as“comprising” is interpreted when employed as a transitional word in aclaim. The descriptions of the various embodiments have been presentedfor purposes of illustration, but are not intended to be exhaustive orlimited to the embodiments disclosed. Many modifications and variationscan be apparent to those of ordinary skill in the art without departingfrom the scope and spirit of the described embodiments. The terminologyused herein was chosen to best explain the principles of theembodiments, the practical application or technical improvement overtechnologies found in the marketplace, or to enable others of ordinaryskill in the art to understand the embodiments disclosed herein.

What is claimed is:
 1. A system, comprising: a memory that storescomputer executable components; and a processor that executes thecomputer executable components stored in the memory, wherein thecomputer executable components comprise: an image decompositioncomponent that decomposes an input image into sub-images; aharmonization component that harmonizes the sub-images withcorresponding reference sub-images of at least one reference image basedon two or more different statistical values respectively calculated forthe sub-images and the corresponding reference-sub images, resulting intransformation of the sub-images into modified sub-images images; and atraining component uses the modified sub-images as ground-truth trainingsamples to facilitate training one or more sub-image harmonizationmodels to transform the sub-images into the modified sub-images.
 2. Thesystem of claim 1, wherein the computer executable components furthercomprise: a reconstruction component that combines the modifiedsub-images into a harmonized image, the harmonized image having a moresimilar appearance to the at least one reference image relative to theinput image.
 3. The system of claim 1, wherein the sub-images compriseenergy band images and the reference sub-images comprise correspondingreference energy band images.
 4. The system of claim 3, wherein the twoor more different statistical values are selected from a groupconsisting of: means of energy band image intensity values of the energyband images and the corresponding reference energy band images, standarddeviations of the energy band image intensity values, percentiles of theenergy band image intensity values, and histograms of the energy bandimage intensity values.
 5. The system of claim 1, wherein the imagedecomposition component generates the sub-images from a low pass signal.6. The system of claim 1, wherein the computer executable componentsfurther comprise: a reference image selection component that selects theat least one reference image from candidate reference images based on adegree of similarity between a first feature vector for the input imageand a second feature vector the at least one reference image.
 7. Thesystem of claim 6, wherein the first feature vector and the secondfeature vector are respectively based on at least one statistical valueof the two or more different statistical values.
 8. The system of claim7, wherein the sub-images comprise energy band images and the referencesub-images comprise corresponding reference energy band images, andwherein the two or more different statistical values are selected from agroup consisting of: means of energy band image intensity values of theenergy band images and the corresponding reference energy band images,standard deviations of the energy band image intensity values,percentiles of the energy band image intensity values, and histograms ofthe energy band image intensity values.
 9. The system of claim 6,wherein the first feature vector and the second feature vector arerespectively based on output features generated based on application ofa deep learning neural network model to the input image and the at leastone reference image, and wherein the deep learning neural network modelwas trained on a corpus of images related to the input image and the atleast one reference image.
 10. The system of claim 6, wherein thecomputer executable components further comprise: a vectorizationcomponent that generates feature vectors for candidate reference images;and a reference image set generation component that selects a subset ofthe candidate reference images based on differences between the featurevectors, wherein the reference image selection component selects the atleast one reference image from the subset.
 11. The system of claim 10,wherein the computer executable components further comprise: aclustering component that clusters the candidate reference images intodifferent groups based on the differences between the feature vectors,and wherein the reference image set generation component selects onecandidate reference image from each of the different groups forinclusion in the subset.
 12. The system of claim of claim 2, wherein thetraining component uses the harmonized image and additional harmonizedimages generated in a same manner as the harmonized image as additionalground-truth training samples to facilitate training a full-imageharmonization model to transform input images into harmonized imagesthat have a more similar appearance to the at least one reference imagerelative to the input images.
 13. The system of claim 12, wherein thefull-image harmonization model comprises one or more deep learningneural network models.
 14. The system of claim 12, wherein the computerexecutable components further comprise: a model application componentthat applies the full-image harmonization model to transform a new inputimage into a new harmonized image.
 15. The system of claim 1, whereinthe one or more sub-image harmonization models comprise a deep learningneural network model.
 16. The system of claim 1, wherein the computerexecutable components further comprise: a model application componentthat applies the one or more sub-image harmonization models to transformnew sub-images decomposed from a new input image into new modifiedsub-images; and a reconstruction component that combines the newmodified sub-images to generate a new harmonized image having a moresimilar appearance to the at least one reference image relative to thenew input image.
 17. A system, comprising: a memory that stores computerexecutable components; and a processor that executes the computerexecutable components stored in the memory, wherein the computerexecutable components comprise: an image decomposition component thatdecomposes an input image into sub-images; a harmonization componentthat harmonizes the sub-images with corresponding reference sub-imagesof at least one reference image based on two or more differentstatistical values respectively calculated for the sub-images and thecorresponding reference-sub images, resulting in transformation of thesub-images into modified sub-images images; a reconstruction componentthat combines the modified sub-images into a harmonized image, theharmonized image having a more similar appearance to the at least onereference image relative to the input image; and a training componentthat uses the harmonized image and additional harmonized imagesgenerated in a same manner as the harmonized image as ground-truthtraining samples to facilitate training a full-image harmonization modelto transform input images into harmonized images that have a moresimilar appearance to the at least one reference image relative to theinput images.
 18. The system of claim 17, wherein the sub-imagescomprise energy band images and the reference sub-images comprisecorresponding reference energy band images.
 19. The system of claim 17,wherein the full-image harmonization model comprises one or more deeplearning neural network models.
 20. The system of claim 17, wherein thecomputer executable components further comprise: a model applicationcomponent that applies the full-image harmonization model to transform anew input image into a new harmonized image.
 21. The system of claim 17,wherein the training component uses the modified sub-images asadditional ground-truth training samples to facilitate training one ormore sub-image harmonization models to transform the sub-images into themodified sub-images.
 22. The system of claim 21, wherein the one or moresub-image harmonization models comprise a deep learning neural networkmodel.
 23. The system of claim 21, wherein the computer executablecomponents further comprise: a model application component that appliesthe one or more sub-image harmonization models to transform newsub-images decomposed from a new input image into new modifiedsub-images from harmonized images; and a reconstruction component thatcombines the new modified sub-images to generate a new harmonized imagehaving a more similar appearance to the at least one reference imagerelative to the new input image.
 24. A system, comprising: a memory thatstores computer executable components; and a processor that executes thecomputer executable components stored in the memory, wherein thecomputer executable components comprise: an image decompositioncomponent that decomposes an input image into sub-images; a referenceimage selection component that selects at least one reference image fromcandidate reference images based on a degree of similarity between afirst feature vector for the input image and a second feature vector theat least one reference image; and a harmonization component thatharmonizes the sub-images with corresponding reference sub-images of atleast one reference image based on two or more different statisticalvalues respectively calculated for the sub-images and the correspondingreference-sub images, resulting in transformation of the sub-images intomodified sub-images images, wherein the first feature vector and thesecond feature vector are respectively based on at least one statisticalvalue of the two or more different statistical values.
 25. The system ofclaim 24, wherein the computer executable components further comprise: areconstruction component that combines the modified sub-images into aharmonized image, the harmonized image having a more similar appearanceto the at least one reference image relative to the input image.
 26. Thesystem of claim 24, wherein the sub-images comprise energy band imagesand the reference sub-images comprise corresponding reference energyband images.
 27. The system of claim 24, wherein the first featurevector and the second feature vector are respectively based on outputfeatures generated based on application of a deep learning neuralnetwork model to the input image and the at least one reference image,and wherein the deep learning neural network model was trained on acorpus of images related to the input image and the at least onereference image.
 28. The system of claim 24, wherein the computerexecutable components further comprise: a vectorization component thatgenerates feature vectors for candidate reference images; and areference image set generation component that selects a subset of thecandidate reference images based on differences between the featurevectors, wherein the reference image selection component selects the atleast one reference image from the subset.
 29. The system of claim 28,wherein the computer executable components further comprise: aclustering component that clusters the candidate reference images intodifferent groups based on the differences between the feature vectors,and wherein the reference image set generation component selects onecandidate reference image from each of the different groups forinclusion in the subset.
 30. The system of claim 24, wherein thecomputer executable components further comprise: a training componentthat uses the harmonized image and additional harmonized imagesgenerated in a same manner as the harmonized image as ground-truthtraining samples to facilitate training a full-image harmonization modelto transform input images into harmonized images that have a moresimilar appearance to the at least one reference image relative to theinput images; and a model application component that applies thefull-image harmonization model to transform a new input image into a newharmonized image.
 31. The system of claim 24, wherein the computerexecutable components further comprise: a training component uses themodified sub-images as ground-truth training samples to facilitatetraining one or more sub-image harmonization models to transform thesub-images into the modified sub-images.
 32. The system of claim 31,wherein the computer executable components further comprise: a modelapplication component that applies the one or more sub-imageharmonization models to transform new sub-images decomposed from a newinput image into new modified sub-images from harmonized images; and areconstruction component that combines the new modified sub-images togenerate a new harmonized image having a mores similar appearance to theat least one reference image relative to the new input image.
 33. Asystem, comprising: a memory that stores computer executable components;and a processor that executes the computer executable components storedin the memory, wherein the computer executable components comprise: animage decomposition component that decomposes an input image intosub-images; a reference image selection component that selects at leastone reference image from candidate reference images based on a degree ofsimilarity between a first feature vector for the input image and asecond feature vector the at least one reference image, wherein thefirst feature vector and the second feature vector are respectivelybased on output features generated based on application of a deeplearning neural network model to the input image and the at least onereference image, and wherein the deep learning neural network model wastrained on a corpus of images related to the input image and the atleast one reference image; and a harmonization component that harmonizesthe sub-images with corresponding reference sub-images of at least onereference image based on two or more different statistical valuesrespectively calculated for the sub-images and the correspondingreference-sub images, resulting in transformation of the sub-images intomodified sub-images images.
 34. The system of claim 33, wherein thecomputer executable components further comprise: a reconstructioncomponent that combines the modified sub-images into a harmonized image,the harmonized image having a more similar appearance to the at leastone reference image relative to the input image.
 35. The system of claim33, wherein the sub-images comprise energy band images and the referencesub-images comprise corresponding reference energy band images.
 36. Thesystem of claim 33, wherein the computer executable components furthercomprise: a vectorization component that generates feature vectors forcandidate reference images; and a reference image set generationcomponent that selects a subset of the candidate reference images basedon differences between the feature vectors, wherein the reference imageselection component selects the at least one reference image from thesubset.
 37. The system of claim 36, wherein the computer executablecomponents further comprise: a clustering component that clusters thecandidate reference images into different groups based on thedifferences between the feature vectors, and wherein the reference imageset generation component selects one candidate reference image from eachof the different groups for inclusion in the subset.
 38. The system ofclaim 33, wherein the computer executable components further comprise: atraining component that uses the harmonized image and additionalharmonized images generated in a same manner as the harmonized image asground-truth training samples to facilitate training a full-imageharmonization model to transform input images into harmonized imagesthat have a more similar appearance to the at least one reference imagerelative to the input images; and a model application component thatapplies the full-image harmonization model to transform a new inputimage into a new harmonized image.
 39. The system of claim 33, whereinthe computer executable components further comprise: a trainingcomponent uses the modified sub-images as ground-truth training samplesto facilitate training one or more sub-image harmonization models totransform the sub-images into the modified sub-images.
 40. The system ofclaim 39, wherein the computer executable components further comprise: amodel application component that applies the one or more sub-imageharmonization models to transform new sub-images decomposed from a newinput image into new modified sub-images from harmonized images; and areconstruction component that combines the new modified sub-images togenerate a new harmonized image having a mores similar appearance to theat least one reference image relative to the new input image.